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' 4 This report presents a pre- and posttest evaluation 
of the first program year of the Kamehameha Early Education Prpgram 
(KEEP). Each of the 28 children in the KEEP kindergarten class 
received three tests: the Hechsler Preschool and Primary Scale of 
Intelligence (WPPSI) , the Metropolitan Readiness Test (MET) , and the 
Standard English Repetition Test (SERT) . The testing scores from the 
fall and Spring were compared on the basis of their correlations with 
each other and with other variables, such as socioeconomic factors. 
, Improvement on the MRT was only moderately correlated with changes in 
I.Q* scores. The children with the greatest changes in I.Q. scores 
tended to have had less school experience and came from lower income 
families; The high and low MRT change groups showed significant 
differences on only three of 100 variables, and the three were not in 
any particular group of variables, as were significant variables of 
the high and % low I.Q. change groups. SERT scores were highly 
correlated with all three I.Q. test measures for both pre- and 
posttestings, and were consistently related to ^socioeconomic factors. 
, While SERT scores did significantly increase from fall to spring, the 
change in SERT scores did not correlate with initial scores. 
(Author/JMB) • / 
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The Kamehameha Early Education Program 



.The Kamehameha Early Education Program (KEEP) i^Mresearch and 
development program of The Kamehameha Schoois/Bernice P. Bishop Estate. 
The mission of KEEP is the development^, demonstration, and dissemination 
of methods for Improving the education of Hawaiian and Part-Hawaiian 
children. These activities are conducted at the Ka Na'i.Pono Research 
and Demonstration School, and in public classrooms in cooperation with 
the State Department of Educatiqn. KE£P piojects and activities involve 
many aspects of the educational process, including teacher training, 
curriculum ^development , and child motivation, language, and cognition. 
More detailed descriptions of KEEP 1 s history and operations are presented 
in Technical Reports //1-4. * I ) 
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Abstract 



This report presents a pre-v and posttest evaluation of the first KEEP 
progral year. The first kindergarten class received three tests: the Wechsler 
Preschool and Primary Scale of Intelligence (WPPSI) f ' t^n e ^Metropolitan Readiness 
Test (MRT), and the Standard English Repetition Test" (SERT) . The scores were 
compared on the basis of their correlations with £ach other and with other 
variables, such as socioeconomic factors. Interestingly enough, improvement 
on\he MRT is only moderately correlated with/changes in I.Q. scores. 

The children with the greatest changes /in X«Q« scores tended to have had 
less School experience and came from lowey income families. The high and low 
MRT /change groups showed significant differences on only three oi: 100 varia- 

Les, and the three were not in any pa/ticular group of variables, as were 

* I i » 

significant variables of the high and/ low I,Q. change groups, 

/ / 

SERT scores were highly correlated with all three I.Q. test measures for 
both pre- and posttestings, and were consistently related to socioeconomic 
factors. While SERT scores did significantly increase from Fall to Spring, 



the change in SERT scores did not correlate with initial scores. 



These relationships between te$/ts and the correlations of test scores 

' » ' ■ j ■ - I 

with other variables must be verified and elaborated on with future classes. 
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Technical Report //5 
Pretest and Posttest Results of the Fir^V^KEEP Program Year 1 ' 
* \ t Ronald Gallimore Roland G. Tharp Gisela E. Speidel 

One can consider many dimensions in program evaluation. At present 
educational research tends to be more concerned with^ process analysis and les 

with outcome evaluation.. Determining that. a program did or did not succeed 

> 

at the end of a school year is pften of minimal value since such ef forts ' f re- 
quently neg^fect to report what occuryed in the classroom. 

As important as process analysis is, howjever, it is also important to \ 
place the consequences of any educational effort in broader perspective. 
Despite their limitations, it remains true that standardized tests provide 
one useful perspective. Data presented here are unusually restricted since 
we do not have appropriate control groups on major testing devices. Such 
data could not be collected until we had begun. our work with public schools. 

. . < Pretest and Posttest Data 

Edch of the 28 childreiji at KEEP received the WPPSI in Fall, 1972- Later 
in the Fall of the same yeat, the Metropolitan Readiness Test 
(MRT) was also administered. Graduate students in psychology administered 
both thef tests. „' . 

The means for the Fall and Spring administrations of the WPPSI and. the,* 
MRT are presented in Table 1. The considerable variability within the group 



^Preparation/ ofyihis report was partially supported by the Socio-behavioral 
Research C^tTpT Mental Retardation Research Centner, UCLA. Computing assis- 
tance was Obtained from Health Sciences Computing Facility, UCLA, supported 
by NIH Sp^dial Research Resources Grant RR-3. 
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of 28 kindergarteners reflects the heterogeneity of the class. For example, 
the standard deviations for the Fall Full Scale I.Q. and the MRT raw scares 
were 20.2 and 16.0, respectively. In Spring, the variation in I.Q. scores 
is somewhat less than it was in Fall, but the standard deviation for the ^ 
Spring administration of the MRT is slightly lower (See Table i) . 

> * Table 1 ' % ■ 



1972-73 KEEP Kindergarten Glass Intelligence and Achievement Test Scores 





V 


September and May 








"0. '■' 


Fall 


(Sept. 1972) 


Spring 


(May 


1973) 


WPPSI* 








. / 


Mean Chan; 


.- 


X 


S.D- 


X 


S.D. 


(S.D.) 




(N=28) 


(N=27)* 






Full Scale 


92.1 


' 20.2 


101.6 


13.8 


10.63 












• (13.07)" 


Verbal 


87.5 


19.4 


92.7 


14.7 


6.48 












(12.49) 


Performance 


98.3 


19.6 


110.9 


13.6 


13.78 










(14.33) 



Metropolitan Readiness T^st^ 

X 

• (N=28 



Total Raw Score 
Percentile 



32.3 
1,9.0 



S.Dt 

16.0 
21.7 



X 

(N=28) 
59.6 
60.3 



S.D. 

11.9 
22.3 



(ti<o4) • 

40.73 . 
(18.26) 



*N=27 (one child was nod retested on, the WPPSI because she left school early) 



For both the three I.Q. scores and the MRT, statistically significant 
increases were -obtained from the pre- to the pofct testing (the mean change 
scores are also presented in Table 1). However, the total" group T.Q. and 
achievement test scored conceal an important feature of the results: the change 

scores actually reflect two essentially different populations. Of the 27 

- / ^ 

children retested on the WPPS-L in May, 14 showed an increase in Full Scale 



.I.Q. of 10 or more scale points; the mean increase in I,Q. for these 14 
children, was 20.7. The remaining 13 children^showecLa mean increase of five 
I.Q. points. Eight of this group increased their scores less than ten 
points, and five decreased an average of seven points. 

Table 2 * 

I.Q. Change Scores - High and' Low Change Groups 

^ SEPTEMBER 1972 * MAY 1973 

Lo * v Hi Lo - Hi 

(9 or less pts.)' (10 or, more pts.) 

Change Change Change Change 

• , (N=27) (N=27) 

Full Scale 104.23' 73.27 104.46 .94.55 • / 

» *' 
Verbal 98.46- 71.27 96.08 85.18 



Performance 109.31 



-80.55 112.69 ^ 105.91 



Tabll 2 presents the mean pre- ^and posttest I.Q. scores for the Hi and 
Lo change groups. In September, the differences are large and dramatic: the 
Lo change group at .the time of the pretest is in the average range of I.Q. ; 
scores, while the Hi change group is well below average. At the time of the 
posttest, however, the gap has narrowed to the point that there are no 
statistically reliable differences between the Hi .and Lo change groups. 

It is not possible to rule out regression effects in this case. However, 
since the KEEP children were selected randomly, the threat to validity posed 
~by regression e/fects is corresppndingly reduced. ' Consideration bf^these 
results should nonetheless be undertaken with appropriate caution. 

Dramatic increases also were obtained for the MRT. The ac^tyal pre- arid 

posttest scores of r 12 children are presented in Table 3. We present the 

i - 

actual scores to illustrate that .the changes obtained are not attributable 
to a few extreme scores; the 12 children showed a 4 mean increase on the MRT 
posttest of 45 or more points^ u~ o 



Changes on the MRT were only modestly correlated with changes on the 

t % 
WPPSI. The correlations between the MRT and the Full Scale, Verbal, and , 

Performance are: .32, .32, andf .18 (df=27^, respectively. These insignificant 
correlations are due to^the fact that there was much greater variability in - 
I.Q. change scores than in MRT scores; that is, all children showed improve- 
ment on the MRT, but* only about half of the group improved on the I.Q. Thus 

if we were to examine only children who showed improvement on both, there 

v 

would be a very large correlation between change scores on the two tests. ' 

Again it is prudent to invoke a word* of caution about the problem of 
Agression effects. On any psychological lor educational measure, it must be 
expected that changes will tend to move toward the population mean score. In 
this case, about half of the children tested very low in the Fall. Since there 
is much more "room" to improve than to score lower, the proba||^#ty that a 
"gain" will occur is correspondingly increased. The question always is whether 
any amount of the observed change in scores can be attributed to some factor 
of interest, in th^s case exposure to the KEEP program. We are ^ot prepared 
to draw that conclusion, but it is appropriate to conclude that 0 the average 
performance level of the children in Spring is within statistically normal 
limits, and thus programmatlcally useful. Whether the change from Pall can 
be attributed to our efforts is problematical. Continued use of random' 
sampling, replication of- these gains with subsequent clasaes, specific hypo- 
thesis testing, continuous monitoring of educational process variables, 
am'ong other things can all serve to Reduce the probability fcha$ regression 
feffects are the sole cause of the obtained changes. 



Table 3^ 

Pre-, Post-, and Change MRT^Scores . f or 12 Children 
who increased scores 45 points or more 



September 
(Percentile) 



May 
(Percentile) 



Change 
(Percentile) 



Entering Metro 
27 
08 
13 
03 
08 
08 , 
07 
05 
13 
05 

25 



' 79 
75 
86 
67 
65 
65 
53 
53 
63 
59 
65 
79 
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52 
67 
73 
64 
57 

46 
48 
«50 
54 
54 
54 



Characteristics of Hi and Lo Change Groups 

Those children who shewed the greatest increase in Full Scale WPPSI I.Q. 
* v ^ 

scores came from families in which the combined occupational status score of 
the parents and of the mother alone .was* relatively lower. Also, children who 
showed most I.Q. change tended not to have attended preschool (See Table Q) . 



Table Q 

Significant Correlates of I.Q. Fall Scale Change Scores 

Class 1, 1972-73 
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Parent's Occupation 
Mother's Occupation 
Preschool 



-.47 (n = 27) 
7 -.43 (n = 27)' 
-.40 (n = 25) 



In Table 4, the mean I.Q. Change scores for children attending and not 
attending preschool are presented. Although those who did not attend pVe- 
school showed a significantly greater increase on Full Scale I.Q - .', the "prin- 
cipal contribution to the difference between the* groups appears to be the 
differential score on Performance I.Q. 'N Thus , children who did not attend 
preschool showed a significant increase in Performance I.Q., but not Verbal • 
I.Q. 

, , ' s i 

Table 4 ' 
i ; ■' ■ . 

M^an I.Q. Change Score^foir 

Children Attending and Not Attending Preschool 

" / -*> /' ' . 

preschool - \ , 

Significance of Mean Difference 



J 




Yes 


No 


' df 




Change 
SD 


X 


5.08 
(10.72) 


14.75 
(12.38) 


23 


2.09* 


Change 
SD 


X 


2.39 
(11.22) 


9.08 
(12.21) 


za 


1.43. 


Change 
SD 


X 


7.15 
(13.41) 


19.50 
(12.24) 


• 23- 


2.40** 



* p<.05 (two-tailed) 
**p<.025 (two-tailed) 



e / In summary, the children with the greatest changes in I.Q. scores tended 
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to have had less school^ experience and to come from loWer income families. 

Of thej.00 variables\examined, only <hrie significant correlates of MRT 

change scores were obtainedX The three, were: the degree of child care • 

u f > * 

assumed by older male siblings, the average number of seconds the child 

attended to another child during an initial experience in peer tutoring, and 

the average attentiveness to a peer tutor over three observation sessions. 

Marginal, but? not statistically significant, relationships were obtained for 

age, pretest MRT* score, apd father's responsibility for child care. Children 



ho increased in MRT scores the most tended to be somewhat younger, have 



lower scores on the MRT pretest, and come from families in which the father 
assumes greater responsibility for child car£. 

Unlike I.Q. change scores, the MRT -change scores do not show* any cons is- " 
tent association witlr^particular plasses of variables, $uch as indices of 
socioeconomic status. Indeed, it is difficult to understand why a correlation 
should emerge between MRT change scores and the attentiveness of children to 
a peer tutor: the peer tutor data were obtained late in the< school year, ^ 
represent relatively'b-rief • periods (a total of threfe ten minute sessions)/ 
and were ^collected ip. Substantially the same mjj^er as a d^lly on-task 
observation code. If a relationship had been obtained between the daily oft- 
task observation measure, it~might have been appropriate to interpret the 
correlation between MRT c\(ange scores and attentivenss to a|peer tutor. As. 
it is, it is more prudent to consider the correlation a chance finding in a . 



100 variable matrix. 



The Standard English Repetition Test 



i Repetition Test i ^ 



(SERT-X/ 

The SERT was deviled as a measure 'of Standard English (SE) performance, 
A more detailed presentation of validity and reliability jhas been reported 
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elsewhere '(Day, Boggs, Speidel, Tharp, and Gallimore, 1975). The SERT. uses- 

r r 

. the device pf sj^itence repetition to assess the child's SE performance level. 
Correlations between SERT performance and I.Q. are significant at the begin- 
ning of the school year; the correlations are .51, .53, and .44 (N=28)' for the 

Full Scale, Verbal, and Performance portions of the WPPSI, respectively. ' » 

. . ^ . . «•••-■.■ 

At the end of the school year, posttest scores on the SERT and the rWPPSI 

correlate .70, .75, and .43 (N=27) for Full Scale, Verbal and Performance. * 

r i 
* ■* 
Both the SRRT and the WEPSI were administered in September and May. The in- 

creased association of SERT, and I.Q. scores in May suggests that SE perfor- 
mance has a greater effect on child performance after school entry. The level 
of association at entry into kindergarten is also high, however , confirming 
the expectation that SE performance is a factor in school achievement from 
the beginning. 

Not surprisingly , SERT scores are consistently related to socioeconomic 
.(SES) factors, Including father's employment status*; occupational level of 
\ mother and father, and father's education. Significant correlations, between 
SERT and SES variables were obtained both at the! beginning and ending of the; 
>school year. However, two variables — mother ' s.. employment status and I teducation- 
were correlated onlf with the May posttest SERT scores. 

In September, the average SERt s^ore was 9*18 (SD=5.72). /This value 
represents the mean number of correct repetitions out of a possible 29. In 
.May, the mean score was 13_.61 (SD=6.62), a statistically significant difference 

(j:=7.85, df-27, p'^.001 for correlated samples). The gain in SE^. performance 

* • *" 

Was .48.6 percent, and this in a classroom in which no formal instruction in 

* " c * • / . 

SE or ^punishment for speaking HCE occurred. % . 

Unlike T.Q. and achievement test data, changes in SERT scores over the ^ 
* school year do not correlate with initial scores" (r=.04, df~26) w . That is, 
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the level of .performance displayed by a child upon entry into kidnergarten 
does not relate to increased SE performance as, measured in May. Gains in 

: 

SE* performance May cannot "he attributed to initially higher levels of per- 

' / . \( * 

fpxnjance in September. ,Whatevei~produced -an .increase in SERT scores occurred 

aftef the child began kindergarten.' 

Table. 5 -presents the significant correlates of SERT change scores, as 

well*as SERT pre- ^and posttest J^cores . Unlike the pre- and posttest scores, 

change scores correlate with 6nfy two socioeconomic variables — both parents 

working and mother's .education. Besides pre-, atad posttest I„Q. scores, the 

only other significant correlate of change^ in SERT performance is the degree 

of the mother's responsibility for child ^ape, and, in this case, children 

who improve more on the SERT come from homes, in. which the mother is relatively 

Hess responsible. Thus, children who improve on the SERT have higher I.Q. 

scores, better educated mothers, both parents working, and are. cared for by 

' . . * \ , • , 

persons other than>the mother. The latter is doubtless a function of the 

■' - • 

greater likelihood of the mother working. * , — , 

Table 5 indicates the lack of correlation between -SERT, I.Q., and MRT 

v 

Change Scores. This is true despite the fact that SERT pre- and posttest 
and change scores correlate with pre- and posttest %I. f Q. scores; indeed, on 
the posttest, the SERT and WPPSI scores were even more closely related than 
on the pretest, as reported above. Thus, the children who show change on the 
SERT and the children who show change on the WPPSI are not the same children./^ 

levels ai^ 

not. - The reason for this apparent contradiction can be found in the relation- 

' r . _ 

ship between pretest and posttest scores and change scores. For the WPPSI, , 
changed in.I.Q. are ne gatively related to incoming scores; for the SERT, 
tljere is no relationship between changes and pretest scores. On the posttest, 



Although the two tests are correlated, relative shifts ijl score 
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however, the picture is reversed? in May, there is no relationship between 
I.Q, change scores and posttest scores. On the SERT, change scores are 
correlated with posttest scores, In other words, some students began the 
school year with very low I,Q. scores, and improved to a point indistinguish- 
able from those who had begun with average l,Qr scores, In terms of SE per- 
formance, some students ended the year with significantly higher SERT scores 
although they had been indistinguishable from the rest of the class on the -~ 
pretest. In short, changes in l.Q. and SE performance were obtained by two 
different groups within the first KEEP kindergarten* Those who improved in 
I.Q x . were from predominantly lower income families, while those who improved 
on the SERT showed a slight tendency to be from middle class families, as 
reflected by mother's employment and education. Those who improved 'on the 
SERT were, furthermore, already at or above average l.Q. on both the pre-^ and 
posttest administration of the WPFSI. 

These data suggest that the significant and dramatic increases in I.Q. 
obtained by approximately, -half the class cannot be attributed to increased 
SE performance levels. The language factor did. not hinder significant I.Q. 
gains by those initially at the lower end of the I.Q. distribution. 

The SERT is' scored for several classes of responses. The basic score 
is the number of accurate repetitions of 29 critical grammatical features. 

4 

In addition, the child's response is scored for other forms of SE that are 

* 

substituted and are appropriate linguistically. Moreover, the total SE 

i ■ 

correct and the. total SE substitutions can be combined^ to form a measure of 
Total Appropriate SE. 

In some cases, children respond to the SERT with Hawaii Creole English 
(HCE) substitutions. These HCE substitutions also vary in terms of appropriacy 
in the sense that a child may repeat a sentence using an HCE form that is 
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Table 5 



Signif icant Correlates of Rep. Test and HCE 



Full Scale I.Q. Sept (Pretest) 
Verbal I.Q. Sept. 
Performance I.Q. Sept. 

Full Scale I.Q. May (Posttest) 
Verbal I.Q. May 
Performance I.Q. May 

Full Scale Change 
SERT Test Change 

' - ' V . 

Total On-Task 
Total pages complete 
1 month learning centers 
SE Rep Sept. 
.SE Rep May 

Sample source 
Mother employed 
Parents working 
Parents occup 
Father employed 
Father's occup 
Mother 1 s occup 
Father's educ 
Mother 's educ 



Sept. SERT 
(Pretest) 

'* .51 
.53 
.44 



.56 (27) 
.63 (27) 



.91 



-.45 



,43 
,46 

,42 (25) 

.49 (23) 

,43 (28) 

.57 (23) 



•May SERT 
(Posttest) 

.64 ' .. 

.63 

.58 

.70 (27) 

.76 (27) 

.43 (27). 



^".44 

.42 
.39 

.91 



-.49 
.41 
.53 
.53 

.44 (25) 
.41 (23) 
.54 (28) 
.54 (23) 
.38 (28) 



SERT 
Change 

. .45 
' .40 
.45 



.48 (27) 
.48 (27) 



.44 



.44' 



,51. 



equivalent to an SE form. These are scored as HCE'. appropriate substitutions. 

Finally, by combining Total Appropriate SE and HCE, we obtain a measure 
of Total Language Performance. For heuristic purposes, we can assume that 
this Total Language score reflects, in some degree, the child's language 
performance level, in both dialects— SE and HCE. 

The analysis of the correlations between the SERT. change scores and.; 
these various categories of response to the. SERT provides some further inf or- 
mation on the structure of improvement on the SERT. Those children who im- 
proved on the SERT were more likely to have made HCE substitutions on the 



pretest* (r = .35, df=26). On the posttest, however, there is no j relationship 
between HCE substitutions and SERT change scores. In other WD^Js, children 
who improved on the SERT switched from HCE substitutions on the p|r^te§H^to 
accurate SE repetitions on the posttest. Therefore, it is uncertain whether 
their improvement, is a function of increased familiarity with SE forms, or 
simply, learning to use SE rather than HCE in school settings. The "improve- 

' ' K ! 

ment" on the SERT is probably a case of d'ialect switching cued by j social 
factors, rather than a reflection of language learning. j 

.There are only 'two correlations between SERT changes and socioeconomic^ 
factors/ .These two correlations are toother's educations (r=.51) and parent s, 
employment status (£=.44) . Although posttest scores correlated with all iiine 
SES variables collected, and pretest scores correlated with sip, of the nine, 
change scores ar$ .less consistently related. Compared to thofee who show I.Q. 
change, the children who improve on the SERT are a somewhat: more heterogeneous 
gifoup with respect to SES factors; however, improvement does not alter the 
,rank order of individuals in 'the groups since the v Vetest*and posttest 

" V . v / \- . 1 *" 

correlate very highly (r=:91, df=26) Quite independent of the change score*., 
children vfrio score high ori^the pre- and posttest ^re the same individuals, 
they haye higher i.Q.s and are f rotn.middle-^lasS /families. \ 

Apparently, children -who improved on the SERT came from backgrounds in 
which HCE is an acceptable option. However/ over, the school year the child- 
ren either learn that SE is the dialect ofJthe teachers' or they become more 
-prof icient^inj SE performance. In a n^ event, these *data we have reported 
suggest that dialect usage of entering kindergarteners dpes not predict 
school readiness as measured by I.Q. tests. It shq.uld be noted that there is 
no relationship between SERT change scores and Jthe MRT, which is presumably 
a more direct measure of school readiness than I^Q. 
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Other Variables of Interest 

A number of other variables analyzed provtfS of interest} for'example, 
sex. Boys and girls were compared on 100 variables. There were 13 signifi- 
cant differences, by t test. '' Girls have highe> I.Q. and achievement test 
scores in September ,. higher I.Q. scores in May, 'are on-task more as measured 
by daily observations, completed more pages in learning centers during a one-, 
month sample period, Have fa the rife with more education, and come from homes . £rt £ 
which the father has less caretaklng responsibilities. 

' • . • ' ; . v • 1 . 

• Of the 100 variables, there were only three differences between children 
who attended preschool (n=13) "and those that did not*(n=12). They are^I.Q. 



changes on Full, Verbal and Performance Scales. The fact that these critical 
variables were related to preschool attendance suggests more attention should 
be given to this finding than otherwi should be warranted by three signifi- 



cant dif f-erjertee's 'out of 100 comparisons. 

• v ■ \ ; ■ / -- 

Summary 

The pre- and posttest data suggest that KEEP did more for children whose 
Incoming I.Q. and achievement scores were low and who were from families of 
lower socioeconomic status. Several explanations are plausible. First, the 
program is excellent and works well for children who need it most. Second, 
the kinds of pre- and posttest measures used are highly sensitive- to changes 
in child/adult rapport, and thus, the Increases are largely artifacts; that 

r- ^ ••*•■ 

is, thd&e who scored initially l^were, in fact, capable of higher performance 
but because of social factors obtained depressed scores. - Third; analysis, of > 
vjchangfe scores -is a difficult problem confounded by regression effects. 

" The data do indicate that the KEEP goals were met in an absolute sense. 
The children showed substantial improvement over the program year on meaningful 

t ft 
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performance measutes. Whethier the results suggest that the KEEP approach Is 



relatively superior awaits tjhe collection of appropriate comparative data, 



Implications for >Further Research 



1) Contrcfl groups must [be obtained; 

' ., i ' - ' ■ • 

2) Findings must be verified by future KEEP classes; 

• ■ * • "I ■ j 

3) The. complex relationships apiong improvement , I.Q. , and language 
ability must be studied further. 
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